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When  a  single  job  is  the  target,  established  job  analysis  approaches  provide  relatively 
straightforward  methods  for  identifying  performance  requirements.  However,  when 
multiple  jobs  are  involved,  the  identification  of  a  single  set  of  relevant  performance 
dimensions  can  be  more  daunting.  In  the  application  here,  there  was  also  a  compel¬ 
ling  requirement  to  develop  a  dimension  set  that  conformed  with  U.S.  Navy  fleet  per¬ 
sonnel  perceptions  of  the  performance  domain.  Accordingly,  the  behavioral  perfor¬ 
mance  constructs  that  experienced  Navy  officers  believed  differentiate  effective  from 
ineffective  supervisory  performance  were  gathered  using  a  personal  construct  theory 
protocol.  We  then  used  a  methodology  suggested  by  Borman  and  Brush  (1993)  to  in¬ 
tegrate  and  summarize  these  personal  work  constructs,  to  provide — through  a  series 
of  qualitative  and  quantitative  strategies — a  dimension  set  targeted  toward  supervi¬ 
sors  in  all  Navy  communities.  The  resulting  dimensions  revealed  certain  themes  that 
might  not  have  emerged  if  traditional  job  analysis  strategies  had  been  used.  The  di¬ 
mension  set  and  the  resulting  behaviorally  anchored  rating  scales  appear  widely  rele¬ 
vant  to  Navy  supervisor  jobs,  and  the  performance  appraisal  system  that  employs  the 
scales  is  nearing  implementation.  In  addition,  performance  feedback  and  develop¬ 
ment  tools  were  developed  to  complement  the  performance  appraisal  system. 


Requests  for  reprints  should  be  sent  to  Jerry  W.  Hedge,  Organizational  Solutions  Group,  2701 
Pence  Lane,  Excelsior,  MN  55331.  E-mail:  Jerwhedge@aol.com 
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In  the  fall  of  2000,  an  “Executive  Review  of  Navy  Training”  was  chartered  by  the 
Chief  of  Naval  Operations  (CNO)  to  examine  the  current  state  of  training  in  the 
Navy  and  to  recommend  actions  that  would  lead  to  a  “revolution  in  training.”  One 
result  of  this  review  was  the  formation  of  a  Task  Force  for  Excellence  through 
Commitment  to  Education  and  Learning  (EXCEL).  The  vision  was  to  provide 
clearly  defined  career  paths  and  milestones,  giving  sailors  the  tools  and  opportuni¬ 
ties  they  need  to  grow  professionally  and  personally.  Thus,  Task  Force  EXCEL’s 
mission  was  to  identify  new  ways  the  Navy  can  train,  grow,  place,  and  utilize  per¬ 
sonnel  that  maximizes  their  ability  to  accomplish  the  Navy  mission  and  makes  for 
a  more  productive  and  satisfying  workplace. 


RESEARCH  AND  DEVELOPMENT  OBJECTIVES 

Our  particular  challenge  within  this  broader  framework  was  the  development  of 
performance  management  systems  for  all  Navy  supervisory  and  nonsupervisory 
jobs.  Both  performance  appraisal,  and  feedback  and  development  tools  were 
needed  to  support  the  management  of  Navy  personnel.  Two  primary  objectives 
guided  this  research  and  development  effort.  First,  these  tools  would  need  to  pro¬ 
vide  Navy  supervisors  with  a  mechanism  for  evaluating  performance  and  offering 
performance  feedback  as  well  as  provide  management  with  a  process  for  offering 
long-term  career  development  opportunities.  Second,  it  also  was  deemed  essential 
that  the  project  adopt,  as  a  fundamental  tenet,  fleet  involvement  in  all  phases  of  the 
development  process.  The  notion  here  was  that  previous  performance  appraisal 
initiatives  had  been  “headquarters  solutions,”  with  little  input  from  fleet  personnel. 
Therefore,  a  highly  important  requirement  was  to  develop  a  performance  appraisal 
instrument  that  would  reflect  the  way  experienced  supervisors  in  the  Navy  concep¬ 
tualized  the  performance  requirements  of  Navy  supervisory  and  nonsupervisory 
jobs.  The  belief  among  senior  management,  as  well  as  the  officers  and  enlisted  we 
worked  with,  was  that  a  more  behavior-based  system  reflecting  fleet  values  was 
needed.  As  noted  earlier,  our  efforts  focused  on  developing  performance  manage¬ 
ment  systems  to  cover  all  supervisory  and  nonsupervisory  jobs.  Because  we  used  a 
similar  approach  for  both  types  of  jobs,  we  present  only  the  supervisory  solution 
for  illustrative  purposes. 


APPROACH 

Defining  the  Performance  Domain 

An  important  first  step  in  developing  a  framework  for  coverage  of  the  performance 
domain  for  supervisory  jobs  in  the  Navy  was  to  identify  all  of  the  behaviors  that 
characterize  this  domain  and  to  determine  how  they  are  related.  Subsequently,  it 
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was  necessary  to  reduce  the  universe  of  dimensions  to  a  manageable  number  that 
would  still  represent  the  entire  performance  domain. 

The  process  chosen  was  a  combination  of  a  personal  construct  theory  (Kelly, 
1955)  application  and  a  strategy  recommended  by  Borman  and  Brush  (1993),  in 
their  examination  of  managerial  performance  taxonomies,  and  later  applied  by 
Borman,  Ackerman,  and  Kubisiak  (1994)  in  a  large-scale  study  with  the  Depart¬ 
ment  of  Labor.  The  first  stage  relies  on  experienced  subject  matter  experts  (SMEs) 
to  generate  performance  behaviors  that  differentiate  between  effective  and  ineffec¬ 
tive  supervisors  using  a  modified,  reparatory  grid  approach  (Borman,  1987).  The 
second  stage  requires  that  a  second  group  of  SMEs  categorize  all  these  behaviors 
and  definitions  into  clusters  based  on  content  similarity.  The  final  stage  relies  on 
data  analytic  techniques  to  compare  and  pool  the  category  solutions  generated  pre¬ 
viously  and  then  provides  a  final  performance  category  solution  by  means  of  prin¬ 
cipal  components  analysis  (PC A). 

Briefly,  personal  construct  theory  posits  that,  based  on  their  experiences  over 
time,  individuals  develop  categories  or  dimensions  that  they  use  to  interpret  and 
make  judgments  about  events  or  objects,  especially  other  people.  Personal  con¬ 
struct  theorists  believe  that  these  categories  represent  the  natural  way  that  people 
think  about  their  world,  again,  especially  regarding  other  people  (e.g.,  Adams- 
Webber,  1979).  The  reparatory  grid  protocol  has  provided  a  method  for  individuals 
to  generate  their  personal  constructs  by  contrasting  different  role  persons  (e.g., 
mother,  best  friend).  In  this  application,  we  were  asking  participants  to  generate 
their  personal  constructs  related  to  job  performance,  what  have  been  referred  to  as 
personal  work  constructs,  or  “folk  theories”  of  performance  (Borman,  1987). 
Again,  this  personal  construct  theory  approach  addressed  the  requirement  that  the 
performance  dimensions  be  the  result  of  a  fleet  solution,  with  the  final  dimension 
set  emerging  from  the  way  fleet  personnel  conceptualize  supervisory  job  perfor¬ 
mance  requirements. 

Performance  behavior  generation.  The  first  stage  of  defining  the  perfor¬ 
mance  domain  focused  on  gathering  information  about  the  important  compo¬ 
nents  that  lead  to  superior  performance  by  supervisory  personnel.  Participants 
were  told  that  their  task  was  to  identify  at  least  five  very  effective  and  five  less 
effective  officers  they  had  known  or  served  with  in  the  past  and  to  focus  on  what 
types  of  behaviors  differentiated  pairs  of  these  effective  and  ineffective  officers. 
This  protocol  for  generating  performance-related  behaviors  is  based  on  the 
reparatory  grid  approach  used  by  George  Kelly  and  his  followers  to  generate . 
personal  constructs  (Kelly,  1955). 

Performance  behavior  generation  workshops  were  conducted  at  Naval  installa¬ 
tions  on  both  the  East  Coast  and  West  Coast  of  the  United  States.  Participants  were 
selected  so  as  to  produce  a  diverse  cross-section  of  officers,  with  respect  to  type  of 
duty  assignment  (e.g.,  aviation,  surface,  submarine),  level  of  experience  (i.e., 
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mid-to-high  level),  and  area  of  specialization  (e.g.,  mechanical,  avionics,  medi¬ 
cal).  A  total  of  55  experienced  officers  participated.  The  officers  in  these  work¬ 
shops  averaged  over  16  years  of  active  duty  service  in  the  Navy,  had  worked  in  a 
supervisory  capacity  for  almost  15  years,  and  had  been  at  their  current  command 
for  over  1  year.  Most  of  the  participants  were  White  (78%)  and  men  (76%)  and  had 
a  4-year  college  degree  or  higher.  In  fact,  47%  of  the  participants  had  a  graduate 
degree  of  some  form.  Participants  were  currently  assigned  to  aviation,  surface 
force,  submarine  force,  or  shore-based  activities,  with  the  largest  group  (47%)  cur¬ 
rently  assigned  to  shore-based  activities. 

Performance  behavior  generation  workshop  participants  generated  a  total  of 
529  behaviors.  Each  individual  generated  from  3  to  16  behaviors,  and  on  average 
each  person  wrote  9.62  behaviors.  These  supervisory  behaviors  and  definitions, 
produced  by  workshop  participants,  were  reviewed  and  edited  by  project  person¬ 
nel  to  a  common  format  and  to  reduce  redundancy.  In  addition,  each  behavior  was 
reviewed  for  multidimensionality  and,  where  necessary,  was  separated  into  its  con¬ 
stituent  components  to  make  it  unidimensional.  The  end  result  of  this  review  was 
126  behaviors  covering  the  performance  domain  for  supervisory  jobs  in  the  Navy. 
Table  1  presents  example  behaviors  resulting  from  this  process. 

Sorting  of  performance  behaviors.  The  next  stage  in  defining  the  perfor¬ 
mance  domain  involved  reducing  the  total  behaviors  to  a  smaller  but  representative 
set  of  performance  categories.  This  was  accomplished  by  using  a  sorting  task  that 
asked  SMEs,  working  independently,  to  sort  the  126  behaviors  into  categories  ac¬ 
cording  to  similarity  of  content. 

The  card  sorting  workshops  were  conducted  with  23  officers  at  naval  installa¬ 
tions  in  the  mid-South.  Officers  were  again  selected  with  an  eye  toward  diversity  of 
experience  and  job  type.  Participants  averaged  over  14  years  of  active  duty  service 
in  the  Navy  and  had  held  supervisory  positions  for  almost  12  years.  Workshop  par¬ 
ticipants  were  primarily  White  (83%)  and  men  (74%),  and  most  had  obtained  a 
graduate  level  degree  (52%).  All  of  the  group  members  were  currently  assigned  to 
shore-based  activities. 

Participants  were  instructed  to  sort  the  behaviors  appearing  on  3  x  5  cards  ac¬ 
cording  to  their  similarity  in  content.  Essentially,  participants  were  encouraged  to 
use  the  cards  to  derive  their  own  ideal  performance  measurement  system  for  Navy 
supervisory  jobs.  After  sorting  all  of  the  performance  behaviors  into  summary  cat¬ 
egories,  each  individual  labeled  and  defined  the  categories.  Participants  were  en¬ 
couraged  to  try  to  place  all  of  the  behaviors  into  a  defined  category  but  were  also 
allowed  to  place  them  in  a  “miscellaneous”  or  “does  not  fit”  category  if  necessary. 
They  were  also  encouraged  to  have  between  5  and  15  categories  in  their  system. 

Pooling  of  behaviors  and  PCA.  The  final  stage  in  defining  the  perfor¬ 
mance  domain  involved  collecting  and  comparing  participants’  solutions.  Spe- 
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TABLE  1 

Example  Items  From  Performance  Behavior  Sorting  Workshop 

Ability  to  adapt  to  new  and  changing  missions,  tasks,  and  situations 

Expressing  oneself  in  a  manner  that  is  conducive  to  a  productive  and  harmonious  environment 
Delegating  work  according  to  priority  and  the  abilities  of  the  available  subordinates 
Balancing  self  development  and  training  with  job  completion  such  that  performance  does  not  suffer 
Providing  prompt,  regular,  and  specific  feedback  to  subordinates 

Using  resources  efficiently  and  effectively  to  maximize  benefit  and  minimize  both  short-  and 
long-term  costs 

Effectively  adopting  different  leadership,  styles  as  appropriate  to  individuals  and  settings 
Accepting  responsibility  for  one’s  own  actions  and  the  actions  of  subordinates 
Supporting  Navy  and  Command  missions  and  goals  willingly,  regardless  of  personal  feelings 
Providing  informational  resources  to  ignite  and  stimulate  in  your  subordinates  a  personal  or 
professional  desire  for  self-improvement 

Ability  to  identify  goals,  assess  available  resources,  and  develop  an  effective  plan  to  achieve  a  goal 
Listening  attentively  to  others,  making  them  feel  comfortable  and  valuable  while  conversing, 
whether  giving  direction  or  seeking  information 
Providing  timely  and  relevant  information  up  and  down  the  chain  of  command 
Ability  to  adjust  to  a  rapidly  changing  environment  and  modify  goals  and  objectives  based  on 
emerging  requirements 

Counseling  subordinates  accurately  and  honestly  and  assisting  them  with  self-improvement 
Understanding  how  policies  and  actions  fit  into  the  overall  mission  scheme 
Adapting  to  stress  effectively  and  remaining  calm,  focused,  and  competent  in  stressful  situations 
Persuading,  inspiring,  and  motivating  others,  regardless  of  their  relative  positions  in  the  hierarchy 
Displaying  uncompromising  values,  such  as  honor,  courage,  commitment,  integrity,  and  honesty 
Having  a  thorough  understanding  of  military  regulations  and  initiatives  and  carrying  them  out  in 
accordance  with  Navy  standards 

Displaying  appropriate  courtesies  to  superiors,  peers,  and  subordinates 
Taking  personal  ownership  of  tasks,  displaying  dedication  to  goals  and  standards 
Providing  programs  for  upward  mobility  and  promotion  through  group  and  individual  counsel 
Effectively  building  and  leading  individual  and  team  outcomes 


cifically,  for  each  pair  of  performance  behaviors,  the  proportion  of  participants 
who  sorted  both  behaviors  into  the  same  category  was  computed  via  a  Fortran 
program.  For  example,  if  Behaviors  1  and  2  were  sorted  into  the  same  category 
by  8  of  the  23  officers,  .35  was  placed  in  the  1-2  entry  for  the  proportion  matrix. 
Table  2  shows  the  frequency  distribution  of  the  126  x  126  proportion  matrix  and 
depicts  for  various  proportional  magnitudes  the  frequency  with  which  the  7,875 
item  pairs  were  sorted  into  the  same  category  by  the  23  participants. 

This  Was  followed  by  also  computing  an  indirect  similarity  index,  which  indi¬ 
cated  for  any  behavior  pair  the  degree  of  correspondence  between  each  of  these 
two  behaviors’  patterns  of  similarity  with  all  the  other  behaviors.  When  one  behav¬ 
ior’s  pattern  of  similarities  with  other  behaviors  corresponds  closely  to  a  second 
behavior’s  pattern  of  similarities  with  these  same  other  behaviors,  then  the  indirect 
similarity  correlation  between  these  two  is  high.  When  this  correspondence  be- 
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TABLE  2 

Frequency  Distribution  of  Personal  Construct 
Sorting  Proportion  Matrix 


Range 

Frequency 

% 

.90-.99 

2 

0.03 

.80-.89 

10 

0.13 

.70-.79 

57 

0.72 

.60-.69 

83 

1.05 

.50-.59 

103 

1.31 

.40-.49 

200 

2.54 

.30-.39 

564 

7.16 

.20-.29 

754 

9.57 

.10-.19 

1,384 

17.57 

.01-09 

2,732 

34.69 

0 

1,986 

25.22 

tween  the  two  performance  behaviors’  similarities  with  the  other  behaviors  is 
lower,  then  the  indirect  similarity  correlation  is  lower.  As  an  example,  consider  Be¬ 
haviors  1  and  2  again.  Suppose  part  of  the  proportion  matrix  looked  like  Table  3. 

The  indirect  similarity  correlation  using  just  the  data  shown  would  be  .81  (N  = 
6).  Of  course,  in  the  actual  development  of  the  indirect  similarity  matrix  the  size  of 
these  correlations  was  124.  In  this  way,  the  judgments  of  all  participants  were  cap¬ 
tured  and  pooled. 

These  two  steps  produced  a  similarity  correlation  matrix,  consisting  of  correla¬ 
tions  between  each  pair  of  performance  behaviors,  which  was  then  submitted  to  a 
PCA  with  orthogonal  rotation  of  components  to  a  varimax  solution.  Component  so¬ 
lutions,  ranging  between  5  and  15,  were  analyzed  and  interpreted.  Results  from  the 
PCA  indicated  that  the  eight-component  solution  was  the  most  interpretable.  The 
criteria  for  assigning  a  performance  statement  to  a  category  was  a  loading  of  .50  or 
greater  and  less  than  .20  on  the  remaining  components.  Table  4  depicts  the  final  PCA 
solution.  It  should  be  noted  that  the  PCA  rotated  solution  was  very  clean,  with  the 
mean  factor  loading  for  all  behaviors  on  the  factors  they  defined  equal  to  .75 ,  and  the 
mean  loading  on  the  other  factors  equal  to .  1 4 .  The  eight  performance  categories  that 
resulted  from  the  PCA  were  reviewed,  labels  were  applied,  and  definitions  were 
written  to  clearly  describe  and  differentiate  the  categories. 


TABLE  3 

Sample  Matrix  of  Behaviors  1  and  2 


1 

2 

3 

4 

5 

6 

7 

126 

Behavior  1 

— 

.35 

.61 

.04 

.09 

.50 

.21 

.32 

Behavior  2 

.35 

— 

.42 

.13 

.19 

.35 

.09 

.41 

TABLE  4 

Interpretation  of  the  Eight  Component  Solution 


Label 


Definition 


Coaching/Mentoring 


Resource  Stewardship 


Displaying  Professionalism 
and  Integrity 


Communication  Skills 


Leading  Change 


Leading  People 


Displaying  Organizational 
Savvy 


Embracing  Personal 
and  Professional 
Development 


Providing  guidance  to  subordinates;  assessing  strengths  and  weaknesses  in 
personnel  and  providing  them  with  honest  and  specific  feedback;  designing 
opportunities  for  subordinates  to  develop  new  skills  and  assisting  them  in 
establishing  career  plans;  providing  subordinates  with  strategic  vision  and 
goals;  sharing  knowledge  and  experience  with  subordinates;  creating  a  work 
environment  that  makes  individuals  feel  valued  and  motivates  them  to  excel 
Managing  resources  efficiently  and  effectively;  ensuring  deadlines  are  met 
through  planning  and  effective  communication  of  objectives;  gathering 
information,  identifying  goals,  assessing  available  resources,  and  developing- 
innovative  plans  to  complete  projects  on  time  and  within  budget;  prioritizing 
tasks  and  delegating  work  appropriately;  relating  tasks/assignments  to  the 
overall  unit  mission;  sorting  through  large  quantities  of  information  efficiently 
Accepting  responsibility  for  own  and  subordinates’  actions;  always  maintaining 
ethical  principles  and  telling  the  truth,  regardless  of  consequences;  displaying 
uncompromising  values  (e.g.,  honor,  courage,  commitment,  integrity); 
willingly  undertaking  necessary  actions,  even  when  physically  risky; 
maintaining  sharp  military  appearance  and  physical  health/fitness;  supporting 
Navy  and  Command  missions  and  goals 
Practicing  meaningful  two-way  communication  (i.e.,  speaking  clearly,  listening 
attentively,  and  clarifying  information);  providing  timely  and  relevant 
information  up  and  down  the  chain  of  command;  tailoring  presentations  to 
the  level  of  the  audience;  expressing  opinions  when  appropriate;  expressing 
oneself  in  a  manner  that  produces  a  productive  and  harmonious  environment; 
ability  to  evaluate  the  importance  of  infoijnation  being  communicated 
Being  open  to  new  ideas  and  new  methods  for  accomplishing  goals;  ability  to 
adjust  to  a  rapidly  changing  environment,  and  modify  goals  and  objectives 
based  on  emerging  requirements;  embracing  change  and  looking  for  better 
methods/techniques  to  accomplish  tasks;  ability  to  adapt  to  new  and  changing 
missions,  tasks,  and  situations;  remaining  calm,  focused,  and  competent  in 
changing  or  stressful  situations 

Effectively  building  and  leading  individual  and  team  activities;  persuading, 
inspiring,  and  motivating  others,  regardless  of  their  relative  positions  in  the 
hierarchy;  creating  a  sense  of  enthusiasm  and  purpose  in  own  team; 
demonstrating  a  positive  attitude,  team  spirit,  and  personality  to  inspire 
subordinates;  effectively  adopting  different  leadership  styles  as  appropriate  to 
individuals  and  settings 

Having  a  thorough  understanding  of  military  regulations  and  initiatives  and 
carrying  them  out  in  accordance  with  Navy  standards;  following  policies, 
regulations,  and  orders,  and  defending  them  to  subordinates;  understanding 
the  chain  of  command,  and  accepting  and  respecting  the  decisions  of 
superiors;  displaying  appropriate  courtesies  to  superiors,  peers,  and 
subordinates;  understanding  how  policies  and  actions  fit  into  the  overall 
mission  scheme 

Continuously  improving  professional  skills,  knowledge,  and  abilities  through 
formal  and  informal  training,  off-duty  education,  on-the-job  training,  etc.; 
ability  to  find  purpose,  personal  benefit,  and  growth  in  work;  balancing  self 
development  and  training  with  job  completion  such  that  performance  does  not 
suffer;  maintaining  superior  technical  skills  through  training 
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Developing  a  Behavior-Based  Performance  Appraisal  Tool 

After  the  performance  domain  for  supervisory  Navy  jobs  had  been  defined,  the 
next  step  was  to  transform  the  performance  categories  that  covered  the  domain  into 
a  set  of  performance  management  tools.  Although  a  variety  of  approaches  was 
possible,  we  felt  it  was  important  to  produce  a  behavior-based  system  so  that  su¬ 
pervisors  using  the  system  could  base  both  their  evaluation  and  their  feedback  on 
concrete  behaviors.  Of  course,  we  already  had  the  126  behaviors,  helping  to  define 
performance  on  the  eight  categories.  However,  it  seemed  important  to  have  addi¬ 
tional  behaviors  to  define  especially  the  low  and  mid-range  levels  of  effectiveness 
for  each  of  the  categories  (the  126  behaviors  were  all  positively  worded).  The  criti¬ 
cal  incident  method  (Flanagan,  1954)  appeared  ideally  suited  for  supplementing 
the  behavioral  content  for  the  categories  as  well  as  for  encouraging  fleet  involve¬ 
ment  in  the  development  process. 


Critical  incident  generation.  This  task  involved  asking  experienced  officers 
to  describe  examples  of  performance  that  represented  the  eight  categories.  Another 
group  of  participants  was  selected  so  as  to  produce  a  diverse  cross-section  of  offi¬ 
cers,  with  respect  to  type  of  duty  assignment,  level  of  experience,  and  area  of  spe¬ 
cialization.  The  critical  incident  generation  workshop  was  conducted  with  21  offi¬ 
cers  at  a  naval  installation  on  the  East  Coast.  Workshop  participants  averaged 
almost  18  years  of  active  duty  service  in  the  Navy,  had  supervisory  responsibilities 
for  almost  15  years,  and  had  been  at  their  current  command  less  than  2  years.  They 
were  primarily  White  (71%)  and  men  (91%)  and  had  attained  at  least  a  4-year  col¬ 
lege  degree.  Approximately  52%  of  them  were  assigned  to  shore-based  activities, 
and  38%  were  assigned  to  surface  force  activities. 

Workshop  participants  produced  161  critical  incidents.  The  critical  incidents 
were  reviewed  by  project  personnel  for  clarity  and  edited  to  a  common  format.  In 
addition,  in  conversations  with  a  number  of  workshop  participants,  and  other  Navy 
project  personnel,  it  was  suggested  that  “Mission  Accomplishment”  was  seen  as 
an  important  “bottom- line”  component  of  performance  that  was  represented  in  all 
the  dimensions  of  performance  but  should  be  highlighted  as  a  separate  dimension. 
On  further  discussion  and  review  of  the  content  of  the  critical  incidents,  this  per¬ 
spective  was  confirmed,  and  it  was  decided  that  a  ninth  performance  category 
should  be  added  to  our  set  of  dimensions  representative  of  the  supervisory  perfor¬ 
mance  domain. 

Adopting  the  behavior  summary  scale  approach  developed  by  Borman  (1979), 
the  critical  incidents  were  content  analyzed  within  each  performance  category,  and 
behavioral  summary  statements  were  written  that  reflected  the  important  themes 
for  that  category  at  the  high,  mid-range,  and  low  effectiveness  levels.  These  27 
summary  statements,  then,  became  the  behavioral  anchors  for  the  scales. 
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Anchor  retranslation.  A  critical  next  step  was  to  verify  that  these  27  behav¬ 
ioral  summary  statements  written  to  anchor  the  three  effectiveness  levels  for  each 
of  the  nine  performance  categories  represented  the  behaviors  associated  with  the 
intended  performance  category  and  effectiveness  level.  This  was  accomplished  by 
means  of  an  anchor  retranslation  process. 

The  anchor  retranslation  workshop  was  conducted  with  1 1  officers  at  a  naval  in¬ 
stallation  on  the  West  Coast.  Again,  selection  of  participants  was  based  on  a  desire 
for  diversity  of  experience.  These  officers  averaged  almost  16  years  of  active  duty 
service  in  the  Navy  and  had  14  years  of  supervisory  experience.  Eighty-two  per¬ 
cent  of  the  participants  were  White,  and  82%  were  men.  Most  had  a  graduate  de¬ 
gree.  Fifty-five  percent  of  the  participants  were  assigned  to  shore-based  activities, 
27%  were  assigned  to  aviation  activities,  and  the  remainder  was  assigned  to  either 
submarine  or  surface  force  activities. 

Participants  were  asked  to  read  through  each  of  the  27  summary  statements,  place 
them  into  a  relevant  performance  category,  and  then  rate  the  effectiveness  of  the  be¬ 
haviors  described  in  each  performance  summary  statement  as  high,  mid-range,  or 
low  effectiveness.  Overall,  there  was  a  very  high  level  of  agreement  across  raters  in 
terms  of  placement  of  anchors  in  categories  and  effectiveness  levels.  In  99.2%  of  the 
cases,  the  officers  sorted  the  performance  statements  into  the  intended  category  and 
effectiveness  level.  An  example  of  the  rating  scale  for  the  “Coaching/Mentoring”  di¬ 
mension  for  supervisory  jobs  is  presented  in  Figure  1. 

Developing  a  Performance  Feedback  and  Development 
System 

The  performance  appraisal  tool  developed  for  supervisory  jobs  was  designed  to 
provide  a  mechanism  for  formal  performance  evaluations  annually.  However,  to 
benefit  fully  from  this  service-wide  initiative,  attention  also  needed  to  be  focused 
on  improving  day-to-day  performance  on  these  important  categories  that  comprise 
the  job  domain. 

Consequently,  we  developed  a  set  of  performance  feedback  forms  that,  when 
linked  to  the  important  performance  categories  in  the  appraisal  tool,  provided  per¬ 
formance  themes  for  both  supervisors  and  job  incumbents  to  target  for  develop¬ 
ment.  To  accomplish  this,  we  first  returned  to  the  performance  statements  (126  for 
supervisory  jobs)  and  the  critical  incidents  (161  for  supervisory  jobs)  that  served 
as  the  raw  material  for  subsequent  development  of  the  behavior  summary  scales. 
We  then  extracted  relevant  dimension-specific  performance  themes. 

These  themes  were  used,  along  with  the  behavior  summary  scale  anchors,  to 
provide  a  more  detailed  list  of  behaviors  for  the  feedback  system.  The  rationale 
here  was  that  we  wanted  supervisors  conducting  feedback  sessions  to  have  as  rich 
a  behavioral  depiction  of  each  dimension’s  domain  as  possible  to  provide  maxi¬ 
mally  relevant  feedback  to  job  incumbents.  Each  performance  category  is  defined, 
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Providing  guidance  to  subordinates;  assessing  strengths  and  weaknesses  in  personnel  and  providing  them  with  honest  and 
specific  feedback;  designing  opportunities  for  subordinates  to  develop  new  skills  and  assisting  them  in  establishing  career 
plans;  providing  subordinates  with  strategic  vision  and  goals;  sharing  knowledge  and  experience  with  subordinates;  creating 
a  work  environment  that  makes  individuals  feel  valued  and  motivates  them  to  excel. 


Exceeds 
Acceptable 
Levels  of 
Performance 


Meets 
Acceptable 
Levels  of 
Perfor- 


Fails  to 
Meet 

Acceptable 
Levels  of 
Perfor- 


Frovides  subordinates  with  highly  useful  guidance  to  enhance  their  professional 
development;  very  clearly  defines  expectations  for  subordinates'  assignments  and  then 
provides  timely,  constructive,  and  supportive  feedback  on  performance;  provides  the 
developmental  opportunities  to  encourage  subordinates  to  continuously  leant  and  improve 
their  performance;  takes  a  highly  active  interest  in  subordinates’  professional  growth  and 
career  progression. 


Typically  provides  subordinates  with  useful  guidance  regarding  their  professional 
development,  but  advice  may  at  times  be  off  the  mark  or  otherwise  unhelpful;  usually 
provides  subordinates  with  clear  expectations  about  performance,  but  may  sometimes  not 
give  feedback  on  tasks  or  feedback  may  not  be  constructive;  provides  some  developmental 
opportunities  to  subordinates;  shows  a  moderate  degree  of  interest  in  subordinates’ 
professional  growth  and  career. 


Either  does  not  give  guidance  or  provides  too  much  direction  on  subordinates’  assignments, 
resulting  in  micromanaging  their  performance;  defines  expectations  poorly  regarding 
subordinates’  assignments,  and  feedback  on  subordinate  performance  is  lacking  or 
unhelpful;  provides  little  or  no  developmental  opportunities  to  subordinates;  shows  very 
little  interest  in  subordinates’  professional  growth  or  career. 


FIGURE  1  Example  of  rating  scale  for  Coaching/Mentoring  dimension. 


and  then  target  performance  themes  are  provided.  The  performance  feedback 
forms  provide  concrete  performance  themes  at  a  greater  level  of  specificity  than 
what  is  available  with  the  rating  scale  anchors.  Consequently,  supervisors  are  able 
to  provide  feedback  during  development  sessions  directed  at  specific  targeted  be¬ 
haviors  within  each  performance  category.  An  example  of  the  detailed  perfor¬ 
mance  themes  for  the  “Coaching/Mentoring”  dimension  is  shown  in  Table  5. 
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TABLE  5 

Example  of  Performance  Feedback  System  With  Targeted  Behaviors 
for  Coaching/Mentoring  Dimension 

Coaching/Mentoring  Providing  guidance  to  subordinates;  assessing  strengths  and  weaknesses  in 
personnel  and  providing  them  with  honest  and  specific  feedback; 
designing  opportunities  for  subordinates  to  develop  new  skills  and 
assisting  them  in  establishing  career  plans;  providing  subordinates  with 
strategic  vision  and  goals;  sharing  knowledge  and  experience  with 
subordinates;  creating  a  work  environment  that  makes  individuals  feel 
valued  and  motivates  them  to  excel 

Target  behaviors  Provides  subordinates  with  useful  guidance  on  professional  development 

for  performance  Makes  assignments  to  promote  subordinate  development  and  growth 

Offers  suggestions  and  advice  that  result  in  subordinates’  learning  and 
increased  effectiveness 

Defines  expectations  for  subordinate  performance  and  helps  them  set  goals 
Provides  training  to  subordinates  and  others  when  they  need  it 
Takes  an  active  interest  in  subordinates’  professional  development  and 
career  progression 

Discusses  with  each  subordinate  their  goals  and  career  aspirations 
Checks  on  progress  of  subordinates’  professional  development  and  provides 
advice  and  support,  as  appropriate 
Provides  timely,  constructive  feedback  op  performance 
Provides  developmental  opportunities  for  subordinates 


DISCUSSION 

This  article  detailed  the  activities  undertaken  to  create  a  new  performance  manage¬ 
ment  system  for  supervisory  jobs  in  the  U.S.  Navy.  As  noted  previously,  although 
we  developed  performance  management  systems  in  parallel  for  both  supervisory 
and  nonsupervisory  jobs  (see  Hedge,  Borman,  Bruskiewicz,  &  Bourne,  2002), 
space  limitations  prevented  a  discussion  of  both  systems. 

The  question  might  be  asked,  how  is  the  approach  described  here  to  develop 
these  performance  management  systems  different  from  other  more  traditional  job 
analysis  strategies,  and  are  the  products  (i.e.,  the  category  systems)  really  different 
from  what  would  be  expected  with  more  traditional  approaches?  The  answer  to  the 
first  part  of  the  question  we  believe  is  yes.  The  strategy  of  using  personal  construct 
theory  to  generate  personal  work  constructs  in  the  target  population’s  own  words, 
and  then  capturing  these  officers’  views  of  the  structure  of  constructs  would  seem 
to  provide  in  summary  form  the  fleet’s  perspective  on  the  important  performance 
requirements  and  how  they  should  be  configured.  That  the  retranslation  results 
were  almost  perfect,  considerably  better  than  typically  is  the  case  with  criterion  de¬ 
velopment  using  a  behaviorally  anchored  rating  scale  (BARS)  approach,  rein¬ 
forces  the  notion  that  this  particular  configuration  of  performance  categories  was 
very  well  understood  by  the  officers.  In  fact,  in  the  context  of  personal  construct 
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theory,  these  categories  were  well  differentiated  from  each  other  (Kelly,  1955),  for 
this  population  and  this  differentiation  extended  to  officers’  understanding  of  dif¬ 
ferent  performance  levels  within  each  of  the  categories.  As  mentioned,  in  only  1  of 
297  (27  Behavior  Summary  Statements  x  1 1  Retranslation  Subjects)  cases,  a  sin¬ 
gle  officer  reversed  two  of  the  statements  in  retranslation. 

This  rather  extreme  success  in  retranslation,  compared  to  typical  results  in  BARS 
studies,  implies  that  the  answer  to  the  second  part  of  the  question  might  also  be  yes; 
that  is,  the  dimensional  system  developed  here  might  have  been  different  from  that 
derived  using  BARS  or  some  other  more  traditional  methodology.  However,  it  may 
be  instructive  to  also  take  a  closer  look  at  the  categories  derived  in  this  study  com¬ 
pared  to  the  typical  dimensions  resulting  from  supervisory  job  analysis.  First,  cer¬ 
tainly  communication  skills,  generated  in  this  study,  is  often  seen  as  a  dimension 
emerging  in  supervisory  job  analyses.  Similarly,  Leadership  is  almost  always  a  re¬ 
sultant  criterion  area  for  supervisor  jobs.  In  the  latter  case,  however,  the  leadership 
construct  is  subdivided  into  Leading  People  and  Leading  Change,  arguably  two  dis¬ 
tinct  leadership  performance  constructs,  and  the  Coaching/Mentoring  category  can 
be  viewed  as  athird  leadership  concept,  distinct  from  the  othertwo.  Thus,  the  config¬ 
uration  of  the  leadership  “space”  may  be  somewhat  unusual  and  dissimilar  to  what 
might  have  emerged  using  more  typical  methods .  For  example ,  in  the  previous  N avy 
performance  appraisal  system,  there  was  simply  a  single  “Leadership”  dimension. 
Resource  Stewardship  is  an  interesting  category  in  the  current  system,  in  that  it  in¬ 
corporates  planning,  organizing,  goal  setting,  and  developing  solutions  to  problems, 
but  separates  out  these  activities  in  relation  to  the  organization  and  its  nonperson  re¬ 
sources  from  activities  directly  involving  subordinates  in  the  organization.  Dis¬ 
playing  Professionalism  and  Integrity  may  not  be  unusual  in  content  compared  to 
what  would  be  expected  from  other  analysis  methods. 

The  Displaying  Organizational  Savvy  category  may  be  the  most  unique  and 
nonobvious  concept  in  the  new  system.  It  involves  a  seemingly  sophisticated  view  of 
following  rules,  regulations,  and  policies,  and  integrating  these  activities  into  new 
initiatives  to  move  decisively  toward  mission  accomplishment.  This  may  be  the  best 
example  of  a  construct  well  understood  and  embraced  by  officers  as  an  important 
performance  requirement  but  emerging  only  when  officers  were  asked  to  create  their 
own  view  of  the  performance  domain,  both  with  respect  to  their  personal  work  con¬ 
structs  and  how  those  constructs  should  be  configured  to  create  a  performance  cate¬ 
gory  in  this  area.  Finally,  the  category  Embracing  Personal  and  Professional  Devel¬ 
opment  is  not  unique  with  respect  to  supervisory  dimension  content,  but  the  strong 
themes  of  taking  personal  control  over  own  self-development  and  ensuring  that  de¬ 
velopmental  activities  do  not  get  in  the  way  of  mission  accomplishment  seem  to  re¬ 
flect  a  well  understood  ethic  among  Navy  officers. 

Thus,  in  response  to  the  second  question  about  unique  and  unusual  performance 
category  content  and  configurations,  on  balance,  our  qualitative  analysis  suggests 
some  commonality  with  typical  supervisory  dimensions  but  also  some  uniqueness, 
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especially  regarding  category  configuration.  As  mentioned,  the  strong  retrans¬ 
lation  results  suggest  that  the  category  system  is  central  to  these  officers’  under¬ 
standing  of  Navy  supervisory  performance  requirements. 

In  sum,  the  combination  personal  construct  theory  and  sorting  protocol  strategy 
for  performance  category  development  may  be  considered  as  an  alternative  to 
more  traditional  criterion  development  approaches.  Results  in  this  application 
were  promising  although  it  will  be  important  to  compare  this  strategy  with  more 
traditional  criterion  development  strategies  (e.g.,  critical  incidents  or  BARS,  task 
analysis,  interviews  with  management).  Such  comparisons  might  include 
retranslation  results  (as  appropriate),  psychometric  properties  of  ratings  using  the 
category  systems  (e.g.,  interrater  reliabilities),  and  user  reactions. 
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